Criteria for Software Testing Tool Evaluation – A Task Oriented View

Considering the immense variety of test tools available, both commercial and open source, an extensive evaluation of these tools with respect to their adequacy for a given organizational or project context seems to be impossible. On this account we present a systematic approach for deriving easily verifiable evaluation criteria for test tools. We define both, quality criteria and functional criteria. Using the TORE methodology we identify activities which potentially could be automated or at least supported by a test tool. Starting from these activities we derive evaluation criteria for test tools. We focus on criteria which can be evaluated considering only vendors' instructions, consequently without an extensive laboratory test. The result of our work is a reasonable list of criteria allowing an effective classification and pre-selection of test tools. In a first step we applied our criteria to evaluate three capture & replay tools. At this, our approach proved of value. The fundamental differences between the tools could be identified and additional criteria for this particular test automation technique could be defined.